Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ZVM : New-Generation Type 1.5 RTOS Virtualization Solution (Hypervisor) #84123

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

GueqiXie
Copy link

This PR introduces ZVM (Zephyr-based Virtual Machine), a Type 1.5 hypervisor based on Zephyr RTOS, which allows running multiple virtual machines (VMs) on ZVM.

The sample provides support for running both Zephyr and Linux VMs on ZVM using qemu_max board. To build and run the ZVM, refer to the documentation in this PR for detailed instructions.

The project is led by Professor Guoqi Xie at Hunan University, China, with contributions from the openEuler SIG-Zephyr team and other collaborators.

For more information, refer to the project documentation and the ZVM Main Page https://gitee.com/openeuler/zvm__.

* This needs to be done when the vcpu is created. The step is below:
* 1. Init vtimer and ptimer register.
* 2. Add a timer expiry function for vcpu.
* 3. Add a callbak function.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

callbak -> callback

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, it has been corrected.

What is changed?

- Added a new qemu_max board for QEMU emulation. This includes the
  addition of board-specific files such as Kconfig, defconfig, board.yaml
  and DTS files under `boards/qemu/max`.

- Added support for multi-core SMP with QEMU through the
  `qemu_max_qemu_max_smp` configuration and associated DTS and YAML files.

- Updated SoC-specific configurations under `soc/arm/qemu_max`, including
  Kconfig, SoC-specific settings, and memory regions in `mmu_regions.c`.

- Added device tree include file `qemu-virt-max.dtsi` for enhanced
  compatibility with the qemu_max board.

Why do we need this change?
- The qemu_max board supports the hardware features of ARMv8.1+,
  while the existing qemu_cortex_a53 does not.

Signed-off-by: Guoqi Xie <[email protected]>
1. ARM Virtualization Host Extension (VHE) is a feature introduced in the
ARMv8.1-A architecture to optimize the performance of hypervisors by
allowing them to run at EL2 (the hypervisor exception level) while behaving
like they are running at EL0/EL1 (user/kernel mode) in the guest.
VHE simplifies the implementation of Type-2 (hosted) hypervisors by
reducing context-switching overhead between the host and guest.
We add it for support Zephyr to run on EL2.

2. The VPIDR_EL2 is an ARM system register at EL2 used to provide a
virtualized Processor ID to guest VMs, enabling consistent and
isolated CPU identification in virtualized environments.

Signed-off-by: Guoqi Xie <[email protected]>
ZVM is a hypervisor based on Zephyr, we add arm64
virtulization support with Zephyr.

- Added CMakeLists.txt and Kconfig changes to enable
  conditional inclusion of ZVM-related files when
  CONFIG_ZVM is enabled.
- Introduced offsets.c to define offset symbols for
  ZVM structures, ensuring proper alignment and memory
  layout for ARM64 registers and contexts.

Implemented ARM64 virtualization exception handling
and context switching in hyp_entry.S, hyp_vector.S,
and switch.c:
- hyp_entry.S: save and restore vCPU register contexts
  and handle guest entry/exit logic.
- hyp_vector.S: Defined exception vector tables for
  various exception types at different privilege levels,
  including IRQs and synchronous exceptions.
- switch.c: Implemented ARM64-specific exception handlers,
  virtual interrupt handling, and memory access emulation
  for virtual devices.

- Updated kernel_arch_data.h to include definitions for
  ZVM-specific structures like zvm_vcpu_context_t and vcpu_t.
- Added zvm_offsets_short_arch.h to define macro offsets for
  ZVM-specific structures, facilitating memory layout
  calculations in assembly code.

Signed-off-by: Guoqi Xie <[email protected]>
1. cpu.c:
- Implemented vCPU initialization, including
  system register initialization, context
  saving/restoring, and interrupt handling
  for ARM64 virtualization.
- Added support for vGIC (virtual GIC) and
  virtual timers (vtimers) for ARM64.
- Included context-switching logic for guest
  and host system registers in hypervisor mode (EL2).
- Integrated hardware feature detection to ensure
  virtualization support (e.g., checking for Hyp mode,
  VHE, and GICv3).
- Supported emulation of ARM64-specific system registers
  (e.g., CNTV_CTL_EL0, CNTV_CVAL_EL0) and handling of
  virtual interrupts.

2. timer.c:
- Implemented support for virtual timer initialization,
  including virtual timer context setup and timeout
  mechanisms.
- Added interrupt service routines (ISRs) for virtual
  and physical timers to handle vCPU timer events.
- Integrated logic for emulating guest timer register
  accesses and managing virtual interrupt propagation.
- Included initialization routines for connecting vtimer
  and ptimer interrupts to the Zephyr IRQ subsystem.

- mmu.c: Implemented stage-2 memory management
  for VMs, including mapping/unmapping memory,
  translation table allocation, and memory
  domain management.

Signed-off-by: Guoqi Xie <[email protected]>
1. isr_wrapper.S: add gic eoi deacrive
when process irq.
2. mmu.c: replace sctlr_el1 with sctlr_el2.
3. prep_c.c, smp.c: using arch_set_cpu_id_elx
to set tpidr_el2.
4. reset.c: add support for hcr_el2 and
enalbe VHE, support mpidr_el1, vmpidr_el2,
hstr_el2, mdcr_el2, vttbr_el2.
5. intc_gicv3.c: add deactive support.
6. init.c: add vcpu_struct for thread.

Signed-off-by: Guoqi Xie <[email protected]>
zvm.c:
- Implemented core initialization logic
  for the ZVM module, including hyperviso
  hardware support checks, ZVM management
  structure setup, device initialization,
  and IPI handler.
- Added functions for managing guest VMs,
  such as loading OS images and managing devices.

zvm_shell.c:
- Introduced shell commands for managing VMs,
  including creating, running, pausing, deleting,
  and listing VM information.

- vm_manager.c: Implements VM lifecycle
  management, including creation (zvm_new_guest),
  deletion (zvm_delete_guest), and state
  transitions (e.g., running, paused).
  Adds logic for VM initialization,
  IRQ management, and runtime operations.

- vm.c: Provides core VM handling logic,
  such as inter-VM communication,
  vCPU initialization (vm_vcpus_init),
  and VM state transitions. Implements
  functions for listing VM information
  and handling VCPU IRQs and context.

- vm_cpu.c: Manages VCPU lifecycle,
  including state transitions (e.g.,
  running, paused) and context switching.
  Introduces support for VCPU scheduling,
  SMP configurations, and CPU affinity
  management.

- vm_mm.c: Handles VM memory management,
  including memory partitioning, dynamic
  allocation, and mapping between guest
  physical addresses (GPAs) and host
  physical addresses (HPAs). Supports
  memory domain creation and DTB handling
  for Linux VMs.

Signed-off-by: Guoqi Xie <[email protected]>
This commit introduces key features for managing
virtual devices and interrupts within the ZVM,
enabling efficient resource allocation and IRQ
management for virtual machines (VMs).

- vm_device.c: Implements functions to add,
remove, and manage virtual devices.
Introduces memory mapping for device regions
(vm_vdev_mem_add) and device-specific interrupt
handling (vm_device_irq_init).
Adds support for handling device emulation
(handle_vm_device_emulate) and passthrough
device unmapping. Also includes callback
mechanisms for device IRQs and the initialization
of board-specific devices.

- vm_irq.c: Implements the virtual interrupt
controller for managing VM interrupts.
Includes functions for creating and initializing
the VM interrupt control block (vm_irq_ctrlblock_create)
and IRQ descriptors (vm_virq_desc_init). Adds vCPU IRQ
handling logic (vcpu_wait_for_irq) and integrates IRQ
initialization for devices. Provides mechanisms for
mapping shared and hardware-specific IRQs across
virtual and physical devices.

Signed-off-by: Guoqi Xie <[email protected]>
Implements essential OS-related functions,
including support for Linux and Zephyr VM.
Features include OS metadata handling (get_os_info_by_type),
virtual memory image loading (load_vm_image),
and OS object creation (vm_os_create).
These enhancements enable seamless guest OS integration
and setup within the ZVM hypervisor.

Signed-off-by: Guoqi Xie <[email protected]>
- vgic_common.c: Implements
  common VGIC functionalities,
  including virtual interrupt configuration,
  type management (virt_irq_set_type, virt_irq_get_type),
  priority handling (vgic_virq_set_priority),
  and synchronization mechanisms (virt_irq_sync_vgic,
  virt_irq_flush_vgic). These enhancements allow
  fine-grained control over virtual interrupts and
  efficient communication between VMs and the hypervisor.

- vgic_v3.c: Adds VGICv3-specific implementations,
  such as initialization (vdev_gicv3_init,
  vgicv3_lrs_init), list register management
  (vgicv3_lrs_load, vgicv3_lrs_save), state
  save/load for interrupts, and support for SGI
  (Software Generated Interrupts) handling
  (vgicv3_raise_sgi). These features provide
  support for ARM GICv3-based interrupt controllers
  in virtualized environments.

Signed-off-by: Guoqi Xie <[email protected]>
- Defined pass-through device structures in `pt_device.h`, including
  `pass_through_device_data` and `pass_through_device_config`, which
  support IRQ configuration and device-specific operations.
- Implemented pass-through device initialization and IRQ handling in
  `pt_device_qemu_max.c`, providing a template for integrating
  pass-through devices with qemu_max boards.

Signed-off-by: Guoqi Xie <[email protected]>
This commit introduces PSCI emulation for managing virtual CPU
states and system power in ZVM. It supports PSCI 0.2, including
functions like SYSTEM_OFF, SYSTEM_RESET, CPU_ON, and CPU_OFF.

Signed-off-by: Guoqi Xie <[email protected]>
1. vpl011: Add a new device driver for the PL011 UART controller.
2. vserial: Add a new device driver for the virtual serial port.

Signed-off-by: Guoqi Xie <[email protected]>
- thread.c: set spsr register to EL2 mode.
Whitout this, thread can not switch when Zephyr
run EL2.
- ipi.c: add zvm ipi handler when ipi occur.
- sched.c: add logic for vCPU thread switch.
Only vCPU need to switch in or switch out that
will call do_vcpu_swap function.

Signed-off-by: Guoqi Xie <[email protected]>
@cfriedt
Copy link
Member

cfriedt commented Jan 18, 2025

@GueqiXie - please read https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html

A single PR with over 11 thousand lines of code is likely not going to receive a lot of attention (or sufficient attention, from another perspective).

Often what I do when I have a large change, is keep the large change in draft mode. And make a separate PR for logical groups of commits.

There is a natural dependency ordering, so this forces the dependency commits to be reviewed first. Once those are merged, the original draft PR can be rebased.

That process can be repeated until the entire changeset has been thoroughly reviewed and merged.

You can follow the README file to boot
ZVM and different VM.
Fix some Compliance Checks problems.
Fix qemu_max//smp defconfig and dts.

Signed-off-by: Guoqi Xie <[email protected]>
@GueqiXie
Copy link
Author

@GueqiXie - please read https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html

A single PR with over 11 thousand lines of code is likely not going to receive a lot of attention (or sufficient attention, from another perspective).

Often what I do when I have a large change, is keep the large change in draft mode. And make a separate PR for logical groups of commits.

There is a natural dependency ordering, so this forces the dependency commits to be reviewed first. Once those are merged, the original draft PR can be rebased.

That process can be repeated until the entire changeset has been thoroughly reviewed and merged.

Thank you for your suggestion. I will look into whether there is a better way to structure the PR. The reason I included all the code in a single PR is that these parts are all core components for implementing virtualization in Zephyr. Without any of them, the virtualization functionality cannot be properly validated.

@kartben
Copy link
Collaborator

kartben commented Jan 19, 2025

Thank you for your suggestion. I will look into whether there is a better way to structure the PR. The reason I included all the code in a single PR is that these parts are all core components for implementing virtualization in Zephyr. Without any of them, the virtualization functionality cannot be properly validated.

You need to flip this around: virtualization might need all the commit but many of them are likely useful on their own, so you can work on submitting them separately ; for example, the introduction of qemu_max board

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants